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1. INTRODUCTION 


In the last decade there has been a great deal of research xn 
the area known as software engineering. Two goals of this work 
have been the improvement in quality of the programs written and a 
gam in programmer productivity (however measured) . Much of the 
resulting research has concentrated on better programming 
methodologies . 

In the commercial area a gain m both programmer productivity 
and program quality has been achieved by the use of so-called 4th 
generation languages, such as Mantis, Nomad, etc. These languages 
are largely nonprocedural in nature and succeed because they 
restrict themselves to a fairly restricted and well understood 
domain. Other examples of successful application generators 
include screen generators, statistical packages, spreadsheet 
packages, etc. 

It is our thesis that parser generators are another useful, yet 
often neglected, application generator. In the last ten years 
parser generators have become generally available on a large 
number of computers. A partial list of these would include BOBSW 
[Berger 1978], Lila, LR [Wetherell 1981], LRParse, MetaWare (Tm) 
[DeRemer 1981], Mystro [Collins 1980], and YACC [Johnson 1979], 


We intend to show a number of applications of table-driven 
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parsers, excluding compilers. These applications are not 
especially novel, but do not appear to be widely known. We hope 
to show that table-driven parsers can profitably be used by 
programmers with only a cursory understanding of the underlying 
theory. We do assume that the reader is familiar with 
context-free (BNF) grammars, or equivalently, syntax charts. 
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2. BASIC NOTIONS OF PARSING 

The primary use of parsers has traditionally been in the 
so-called "front-end" of compilers, in which the parser is 
responsible for recognizing the basic constructs of the language, 
including statements, expressions, etc. Basic compiler theory, 
including parsing, is well covered in the texts [Aho 1977], 
[Barrett 1979], and [Waite 1983], while an introduction to just 
(LR) parsing is given in [Aho 1974], However, for our purposes 
the reader need only understand the basic grammar and parsing 
material in these references, and not the details of parser 
construction (for example, sections 1-3 of [Aho 1974]). 

A dictionary definition of the verb "parse" is: 

To resolve into its elements, as a sentence, pointing 
out the several parts of speech and their 
interrelation; to analyze and describe grammatically, 
as a word. 

For our purposes a parser is merely a grammar-based pattern 
recognizer . 

Consider the following BNF grammar for computing arithmetic 
expressions in "+" and with the usual precedence and 
associativity: 
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<calc> 

• • s 

<expr> 

<end> 

<expr> 

: : = 

<expr> 

+ <term> 

<expr> 

: : = 

<term> 


<term> 

; • = 

<term> 

* <factor> 

<term> 

: : = 

<factor> 

<factor> : 

:= ( <expr> ) 

<factor> : 

:= <number> 


The terminal symbol <end> denotes the end of the expression, while 
the terminal symbol <number> denotes an arbitrary number. 

Grammars are useful not just for defining what sentences or 
strings are legal in a language, but more importantly because a 
derivation or parse tree imposes a definite structure on legal 
strings. In a parse tree each nonterminal serves as the root of a 
subtree, where the subtree is derived from one of the productions 
or rules for that nonterminal. A given string is a legal sentence 
of the language if it has at least one tree derivable from the 
goal or start symbol. The grammar is ambiguous if there exists 
some string with two distinct parse trees. 

Consider the string "5 + 4 * 3". The only parse tree for this 
string according to the above grammar is: 
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<calc> 


<expr> 


+ 

1 

<expr> 

1 

+ 

1 

+ 

1 


1 

+ 

1 

<term> 

1 


1 

1 

+ 

1 

+ 

<term> 

1 

1 

1 

1 

<term> 

* 

<factor> 

<f actor> 

1 


1 

1 

<factor> 


<number> 

<number> 

=5 

1 

<number> 

=4 


=3 

In bottom up or LR 

parsing, the 

input is recognized from 


lef t-to-right and the tree constructed from the bottom to the top. 
Thus, first, the <number> 5 is reduced to <factor>, then the 
<factor> reduced to a <term>, etc. After the seventh reduction 


<term> ::= <term> * <factor> 


the tree would appear as: 


<expr> + <term> 


I 

I 


1 

<term> 

i » 

1 1 

t 

1 

i 

<term> * 

<factor> 

<factor> 

1 

1 

1 

<factor> 

<number> 

<number> 

=5 

1 

<number> 

=4 

=3 


In order to be able to use an LR parser. 


one need not be 


familiar with the underlying theory. It is important to know only 
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the order in which the reductions or rules will be applied in 
recognizing the input. In using an automatically generated 
parser, it is usually a simple matter to construct a sample input 
and to have the parser print out the order in which symbols are 
recognized and rules applied. Additionally, of course, a 
programmer must have some familiarity with grammars and have a 
parser generator available. 

In addition, a programmer must know that the front end of a 
compiler is divided into three phases [Aho 19771 : scanner, 
parser, and semantics phase. The scanner is responsible for 
reading the input and coalescing characters into tokens, including 
keywords, identifiers, numbers, etc. Tokens are passed m a 
stream to the parser, which in turn constructs a parse tree of the 
input. As the end of a rule is recognized, the parser invokes the 
semantics routine, passing as arguments the number of the rule 
recognized and the "semantics" associated with each symbol m the 
right hand side of the rule. 

In our university, parser-based programs are commonly built by 
students who have had neither a course m compiler construction 
nor even a course m programming languages (CS15 and CS8, 
respectively, in Curriculum 78 [Austmg 1979] ) . Such programs 
include constructing an assembler in computer organization, a 
simulator m computer architecture, a database query language m 
database systems, etc. 
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More generally, table-driven parsers have been used in: 

1. Compilers 

a. Parse phase 

b. Code optimization 

c. Code generation [Glanville 1978; Ganapathi 1982] 

2. Other translators 

3. Command languages [Campbell 1984] 

4. Editors 

a. Traditional [Noonan 1985] 

b. Language-based [Fischer 1984] 

5. Query languages 

6. Language-based tools 

7. Software design tools 

8. Miscellaneous 

In the sections which follow, we show some typical examples 
from these areas, beginning with query languages. For the sake of 
concreteness, we will show the grammars and associated semantics 
as they would be input to the Mystro system [Collins 1980; Noonan 
1984]. All of these examples were adapted from larger, production 
applications. Most of them could easily be adapted to another 
parser generator. 
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3. QUERY LANGUAGES 

One of the simplest and most popular uses of parsers is in the 
development of interactive, query languages. One example of this 
might be a desk calculator program based on an expanded version of 
the grammar given in the previous section. An added advantage to 
using a grammar is that it forces the developer to explicitly 
specify the user interface, often leading to a simpler and more 
consistent interface. 

However, before proceeding with developing the desk calculator, 
we must explain some of the conventions used. Grammars will be 
given m standard BNP. Each production or grammar rule may be 
followed by zero or more lines of semantics in Pascal [Jensen 
1975] , each of which will be indented slightly. These semantic 
lines may refer directly to grammar symbols, using notation 
adapted from [Aho 1977]; Mystro provides for the translation of 
these references to references to a runtime semantics stack. 

Our completed desk calculator program with appropriate Pascal 


semantics is 

shown 

in 

Figure 1. Three of 

the 

rules have no 

semantics, namely. 

the 

fourth, sixth, and 

last 

rules ; 

this 

results from 

the 

fact 

that the nonterminal 

on 

the left 

will 


automatically inherit any value or meaning associated with a 
single symbol on the right (whether terminal or nonterminal). To 
get a running application, the desk calculator grammar need only 
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be input to Mystro and the resulting parser compiled and linked. 

Mystro requires as additional input a skeletal program or 
skeleton into which is inserted certain generated constants , 
types, Pascal code, etc. These programs are called skeletons 
because they need to be enhanced with application dependent code. 
Over the years a number of distinct skeletons for various 
application areas have been developed. When used with a grammar 
with no semantics, they result in a program that can be compiled 
and run without modification. Such programs are useful in 
debugging the grammar, determining the order in which rules will 
be applied, creating a set of validation tests, etc. 

Table-driven parsers are commonly used as interfaces to 
application programs with complex input command languages, such as 
database query languages. More recently, parser generators have 
been used for the construction of menu-based systems. 
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<calc> <expr> <end> 

wnteln ('= ' , <expr>.val); 

<expr> ::= <expr> + <term> 

<expr>.val := <expr>.val + <term>.val; 

<expr> ::= <term> 

; (* <expr> inherits <term>'s val automatically *) 
<term> ::«= <term> * <factor> 

<term>.val <term>.val * <factor>.val; 

<term> = <factor> 

<factor> { <expr> ) 

<factor>.val := <expr>.val; 

<factor> ::= <number> 

; (* the scanner must provide for <number>.val *) 


Figure 1: Desk Calculator 
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4. MENU-BASED SYSTEMS 

In recent years many interactive systems have used menus rather 
than a more traditional command language. Here again a grammar 
can be used to both specify such an interface, and to implement it 
[Noonan 1985], In some cases the user interface is a very small 
part of the total system and is used partially to hide a very 
large and complex system. In other cases the grammar and 
associated semantic code constitute the entire application. 

An example of the latter type occurs when one computer is used 
to generate batch jobs for another. A menu interface can be used 
to hide the idiosyncracies of the command language of the batch 
computer. In one such system a DEC VAX computer is used to 
generate jobs for a CDC VPS-32 supercomputer; complicating the 
situation is the fact that a CDC Cyber computer is used as a 
front-end to the VPS. Of course, the VAX, Cyber, and VPS all have 
distinct command languages. Thus, scientists on the VAX (who are 
primarily mathematicians) wanting to use the VPS must know three 
command languages! 

In this particular instance, the VAX is used to generate a job 
on the Cyber, which in turn generates a batch job on the VPS. 
Unfortunately, to save output on the Cyber or return it to the 
VAX, the VPS must in turn generate a batch job for the Cyber. 
Thus, a typical job stream contains three batch jobs in two 
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different command languages, consisting of three pages of 
commands . 

In order to minimize this complexity as much as possible, a 
menu-based program was developed, based on a BNF grammar and its 
supporting LR parser. The skeleton used is novel in that it does 
not contain a scanner. Instead the notions of lazy input [Kaye 
1980] and syntactic error recovery are combined. The parser does 
not request any input until it reaches a state in which a token is 
required. At this point, the “error recovery" routine lists on 
the terminal the set of legal input tokens (menu options) and 
reads the user's response. If the user selects a valid menu 
choice, then the error recovery routine returns with the token 
chosen; otherwise the user is reprompted for input. (This 
approach requires the use of "default" reductions [Anderson 1973] 
in the parser tables; fortunately, this is a widely used space 
optimization technique in parser generators.) 

In this particular application, for example, program data can 
exist on the VAX, the Cyber, or the VPS. Usually, the data is 
kept on the VAX if the program is (relatively) fixed and the data 
changes from one run to another. Conversely, the data is usually 
kept on the Cyber if it is large in size and infrequently changed. 
Data is only rarely kept on the VPS due to the limited disk space 
available and the inability to edit VPS files directly. The 
grammar rules defining the data menu appear below: 
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<data_menu> C(yber <cyber_data> <get_unit> <data_menu> 

<data_menu> I (case <icase_data> <get_unit> <data_menu> 

<data_menu> ::= V(ps <vps_data> <get_unit> <data_menu> 
<data_menu> N(o-more 

In this particular implementation, the first letter of each menu 
option is used to select the option; entering a "C" or "c" 

selects the Cyber option. Unfortunately, both VPS and VAX start 
with the same letter, so the location of the VAX machine (ICASE) 
was used instead. The nonterminals <cyber_data>, <icase_data>, 
and <vps_data> are used to prompt for the permanent filename of 
the data file and generate the appropriate batch commands, while 
the nonterminal <get_unit> is used to prompt for the local VPS 
filename or Fortran unit number of the file. Right recursion is 
used in the first three rules to provide a loop. The "N(o-more" 
option is used to exit from this loop. 

A similar, but more complicated structure is used for program 
output files. Each such file can be printed, mailed (via 

electronic mail), or saved. If not mailed, the file can be 
printed or saved on any of the three machines. The grammar rules 
defining these menus appear as follows: 

<output_menu> P(nnt <what_file> <print_menu> <output_menu> 
<output_menu> M(ail <what_file> <mail_file> <output_menu> 
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<output_menu> s: = S{ave <what_file> <save_menu> <output_raenu> 
<output_menu> N(o-more 


<prxnt_menu> 
<prxnt_menu> 
<pr xnt_menu> 
<prxnt_menu> 


C(yber <pr xnt_cyber> 
I (case <print_icase> 
::= V(ps <prxnt_vps> 

: := E(xxt 


<save_menu> 

<save_menu> 

<save_menu> 

<save_raenu> 


::= C(yber <save_cyber> 
::= I (case <save_icase> 
::= V(ps <save_vps> 

: := E (xxt 


In thxs xnstance, the "E(xxt" xs provxded so that a choice can be 
undone, xf desxred. As before, the <what_fxle> nontermxnal 
prompts for the local fxlename or Fortran unxt number of the 
output fxle. The "prxnt" nontermxnals (excludxng <pr int_menu>) 
generate the approprxate batch commands. The "save" nontermxnals 
(excludxng <save_menu>) prompt for permanent fxlenames and 
generate the approprxate batch commands. 

The grammar for thxs applxcatxon xs about half the sxze of a 
grammar for Pascal, counting the number of terminal symbols, 
nontermxnal symbols, grammar rules, and parser states. A 
prototype was developed rapidly and enhanced as the developers 
better understood the batch command languages and the users 
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requested more options. 

Such menu-based interfaces are merely special cases of systems 
which implement a transition diagram. One interesting example is 
the development of an intelligent cockpit aid for aircraft 
[Collins 1985]. For such applications a translator program named 
TD Converter was constructed which permits stylized natural 
language in describing the transition diagram, which is then 
converted automatically to a grammar. TD converter was, of 
course, built using a grammar-based parser. 



5. TRANSLATORS 


Another common application of parser generators is to build 
translators of various kinds, including compilers. For example, 
students at our university often are required to build an 
assembler m a course on computer organization. The only exposure 
of these students to grammars or syntax charts is limited to 
learning Pascal; despite this, they use a parser generator to 
construct their assemblers. We have found this approach to be 
superior to both the manual construction of assemblers and to the 
use of meta-assemblers [Collins 1983]. 

Assembly languages are quite distinct in structure from high 
level languages, and thus, pose distinct, but easily solved 
problems. First, assembly languages typically have a single 
statement per line, with explicit continuation conventions. This 
problem is easily solved by having the grammar represent the 
structure of only a single statement rather than the entire 
program, with the parser being called once per statement. One 
advantage of this approach is that it simplifies syntactic error 
recovery. Explicit continuation of statements across line 
boundaries is easily handled in the scanner. 

A second problem is that assembly languages are typically fixed 
format, with the following general form; 
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<label> <opcode> <operands> <comment> 

Sometimes the opcode, operands, and optional comment are 
constrained to start in fixed columns. However, this problem is 
again easily handled in the scanner. 

A problem with several distinct solutions is that traditional 
assembly languages have no reserved words, yet knowledge of the 
specific opcode is often crucial to further processing of the 
statement. One approach often used m PL/I grammars is to allow 
opcode names to appear explicitly in the grammar, and also to map 
the nonterminal <name> into both the terminal symbol <ident>, as 
well as any otherwise "reserved" names: 

<operand> ::= <name> 

• • • 

<name> ::= <ident> 

<name> : : = LOAD 
<name> ::= STORE 


An alternative approach is to map the opcodes and pseudo ops into 
angled terminals m the scanner: 
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<opcode> = <MACRO> 
<opcode> ::= <LOAD> 
<opcode> = <STORE> 


Yet another approach is to map the opcodes into classes in the 
scanner based on the number and type of operands it requires. 
Although all these approaches will work, we prefer the last one 
since it results in a significantly smaller grammar and 
corresponding parse tables. 

The last problem is that the number and type of operands depend 
on the opcode. For example, IBM 370 instructions [Struble 1975] 
are divided into storage classes, called register-register (RR) , 
register-storage (RX) , another register-storage (RS) , 

storage-immediate (SI), and storage-storage (SS) . The 

opcode-operand formats for each of these classes is as follows: 

<instr> ::= <RR_op> <register> , <register> 

<instr> ::= <RX_op> <register> , <storage> 

<instr> ::= <RS_op> <register> , <register> , <storage> 
<instr> <SI_op> <storage> , <immediate> 

<instr> ::= <SS_op> <storage> , <storage> 

We can combine all of these notions to produce a grammar for (a 
part of) the assembly language for the IBM 370, as shown in Figure 
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2. To this grammar we have added some simple semantics to update 
the program counter, to define labels in pass 1, to generate code 
in pass 2, etc. One of the advantages we have found to this 
approach is that grammars are inherently structured and modular. 
Thus, it is a simple matter to start with a subset grammar, 
validate it on a subset of the test cases, and then expand either 
the grammar or the semantics to include more and more of the 
language being translated. 

Other translators that have been developed using a parser 
generator include a code generator language [Donegan 1979], the 
UNIX (Tm) make utility [Feldman 1979], a database machine 
interface [Fishwick 1983], a utility for typesetting mathematics 
[Kernighan 1975], etc. 
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<bal> <stmt> <eoln> 
pc := NewPC? 

<stmt> ::= <comment> 

<stmt> ::= <label> <mstr> 
if pass = 1 then 
begin 

NewSymbol (symbol, pc, <label>. idname) ; 

Def ineSymbol (symbol) ; 
end; 

<stmt> ::= <instr> 

<instr> <RR_op> <register> , <register> 

begin 

NewPC := pc + 2; 
if pass = 2 then 

GenerateRR (<RR_op>. opcode, <register-l>. register, 
<register-2>. register) ; 

end; (* RR *) 

<mstr> ::= <RX_op> <register> , <storage> 
begin 

NewPC := pc + 4; 
if pass = 2 then 

GenerateRX (pc, <RX_op>. opcode, <register>. register , 
<storage>. index, <storage>.base, 
<storage>. address) ; 

end; (* RX *) 
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<instr> ::= <RS_op> <register> , <register> , <storage> 
begin 

NewPC := pc + 4; 
if pass = 2 then 

GenerateRS (pc, <RS_op> .opcode, <register-l> . register , 
< reg is ter-2>. register , 

<storage>.base, <storage>. address) ; 

end; (* RS *) 

<instr> ::= <SI_op> <storage> , <immediate> 
begin 

NewPC : = pc + 4 ; 
if pass = 2 then 

GenerateSI (pc, <SI_op>. opcode, <immediate> . val , 
<storage>.base, <storage>. address) ; 

end; (* SI *) 

<instr> <SS_op> <storage> , <storage> 
begin 

NewPC := pc + 6; 
if pass = 2 then 

GenerateSS (pc, <SS_op>. opcode, <storage-l> . length, 

<s to rage-1 >. base, <s to r age-1 >. address , 
<storage-2>.base, <storage-2>. address) ; 

end; (* SS *) 

<instr> ;:= <unknown_op> 

error (IllegalOpcode, <unknown_op>. ldname) ; 


<register> 


<number> 



- 22 - 


if pass = 2 then 

<register>. register := <number>. val; 

<storage> ; := <ident> 
if pass = 2 then 
begin 

FindSymbol (<storage>, <ident> . ldname) 
<storage>.base := BaseReg; 

<storage>. index := 0; 
end; 

<storage> <ident> ( <base?> , <index?> ) 
if pass = 2 then 
begin 

FindSymbol (<storage>, <ident>. ldname) 
<storage>.base ;= <base?>. register; 
<storage> . index := <index?>. register; 
end; 

<storage> ::= <ident> ( <base> ) 
if pass = 2 then 
begin 

FindSymbol (<storage> / <ident>. idname) 
<storage>.base := <base>. register; 
<storage>. index := 0; 
end; 

<base?> : : = 

if pass = 2 then 

<base?> . register := BaseReg; 
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<base?> ::= <base> 

<base> ::= <register> 

<mdex?> : : = 

if pass = 2 then 

<index?> . register ;= 0; 

<index?> ::= <register> 

Figure 2: IBM Assembly Language Grammar 
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6. PROGRAMMING LANGUAGE TOOLS 

As noted by Glass [1982], the typical programming environment 
contains very few tools. The average programmer has at his or her 
disposal only compilers, editors, linkage editors, and document 
formatters (or word processors) . However, with the aid of a 
parser generator a given installation can easily improve the 
situation. LALR(l) grammars exist for all the major programming 
languages, including Cobol and Fortran. 

One such parser-based tool is a pretty printer [Oppen 1980], 
which is perhaps most useful in the maintenance phase, 
particularly, of older, heavily modified programs. Another useful 
tool is a cross-reference program, which can easily be developed 
from a simple parser and can be enhanced to compute various 
software metrics, violations of installation standards, etc. 
However, if too many enhancements are wanted, such a project can 
require almost as many resources as the building of a compiler. 
In fact, any set of enhancements which requires the building of a 
complete symbol table for the language being supported may be too 
ambitious for a simple support tool. 

Fortunately, Browne [1978] has shown a simple, yet effective 
way out of this dilemma. For the language under consideration you 
define a set of relations which capture the important semantic 
aspects of the language. Then, a simple table-driven parser can 



-25- 


be used to build a relational database for the program being 
analyzed. The query language for the relational database system 
can then be used to check for violations of installation 
standards, find uses of particular identifiers, produce management 
reports, etc. Such an approach is surprisingly effective. 

Consider the case of building a simple cross referencer for 
Pascal. One of the problems m Pascal is that a name, such as 
"x," may not be unique; there may be several variables, a type, a 
procedure, etc., all named "x." One simple way to handle this is 
to keep a very simple symbol table in the database constructor. 
The symbol table itself can be kept as a stack of binary search 
trees, with one tree per scope level. Scopes are created 
(deleted) by pushing (popping) a level onto (off of) the stack. 
Other than the identifier defined, almost no other information 
need be kept. Then each identifier can be referenced by a (scope, 
identifier) pair, where each scope is given a unique number. 

Similarly, statements cannot be uniquely identified by the line 
they are on, since Pascal allows multiple statements per line. 
It, thus, seems best to uniquely number each statement, as well as 
recording the exact position (line and column) of the statement m 
the program. 

Figure 3 gives a grammar fragment for building some of the 
needed relations. The "define" routines hide the details of the 
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interface to the relational database system used. The 
"iookupname" routine searches the symbol table for the identifier 
and returns the unique scope number for the identifier referenced. 

Other uses of a language parser include augmenting the language 
with some needed facility. These might include adding relational 
database constructs, graphics primitives, etc. Such additions 
would then be mapped into calls to various runtime support 
routines. This approach allows for hiding the complexity of 
dealing with certain subroutine packages. 
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<while_stmt> : := <while_clause> do <stmt> 

<while_clause> ::= <while> <expr> 
begin 

definestmt (stmtno, <while>.lineno, <while>. column, 
whileclause, staticscope) ; 

(* relation stmt(strat#, line#, column#, stmt_type, 

staticscope#) *) 
stmtno := stmtno +1; 
end; 

<while> ;:= while 

usagekind := reference; 

• • • 

<expr> ::= <simple_expr> 

<expr> ::= <simple_expr> <relop> <simple_expr> 

• • • 

<vanable> ::= <ident> 
begin 

lookupname (<ident>.name, scope); 
def menameusage ( <ident>.name, scope, stmtno, 
<ident>.lineno, <ident>. column, usagekind); 

(* relation nameusage (name, staticscope#, stmt#, 

line#, column#, usage_kind) *) 

end; 


Figure 3: Pascal Cross Reference 
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7. CONCLUSIONS 

In our environment we have found the parser generator to be 
extremely useful as a general application generator. It has been 
used to develop both end-user application programs and various 
software tools. We have found that a parser generator can be 
effectively used by programmers having only a knowledge of 
grammars and no training at all in the theory of formal parsing. 

In particular, much of the power of the parser generator has 
been unleashed by the development of skeletal parsers targeted at 
particular application areas. At our university these include: 

— two compiler skeletons, 

— a two pass assembler, 

— a three pass assembler, 

— the query skeleton, 

— the menu skeleton, 

— two skeletons used for code generation. 

We have enhanced these skeletons to facilitate debugging by those 
unfamiliar with parsing theory. Our students have found that the 
time and effort needed to learn to use these tools are well worth 


the investment 
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NOTES 


Metaware is a trademark of Metaware, Santa Cruz, Ca. 
UNIX is a trademark of AT&T Bell Laboratories. 
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